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ABSTRACT 


The  present  paper  deals  with  an  algorithm  for  quadratic  programming 
when  the  matrix  of  the  quadratic  part  of  the  maximand  is  semi-definite.  VVhen 
m  is  the  number  of  linear  inequality  constraints  the  algorithm  leads  to  a 
simplex  tableau  of  order  m  X  2m . 

In  Section  1  an  algorithm  is  given  which  is  valid  when  the  matrix  of 
the  quadratic  part  is  strictly  definite.  It  was  originally  proposed  in  [  1]  , 
but  is  restated  here  in  a  self-contained  way.  Section  2  deals  with 'linear 
programming,  which  is  considered  as  a  quadratic  programming  problem  with 
a  zero  matrix  for  the  quadratic  part  of  the  maximand.  It  is  an  introduction 
to  Section  3  ,  dealing  with  singular  quadratic  programming. 

The  algorithm  solves  explicitly  for  the  dual  problem  as  well. 
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AN  ALGORITHM  FOR  SINGULAR  QUADRATIC  PROGRAMMING 


J.  C.  G.  Boot 


1.  Quadratic  Programming 


The  problem  is  to  maximize  the  function 


(1.1)  Q*(x)  =a'x--^x*Bx  [  S  a  x  -  ^  S  S  b  xx  ] 

i=l  ^  ^  i=l  J=1  ^  ^ 


where  B  is  a  symmetric,  positive  definite  matrix,  subject  to 


n 

(1.2)  C‘x  s  d  [  S  '^hi’^i  “ '^h  ’  h=l,...,m] 


or,  equivalently, 


(1.3)  C'x  +  Xw  =  d,  w^O. 

The  m  -vector  d  will  be  called  the  source-vector;  the  m  -vector  w 
will  be  called  the  slack -vector.  Either  the  slack  is  zero,  and  the 
corresponding  source  is  fully  used;  or  else  the  slack  is  positive,  and 
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the  corresponding  source  Is  amply  available.  Non -negativity  conditions  may 
be  included  in  (1.2)  or  (1.3)  .  The  first  n  inequalities  of  (1.2)  are 
then  -  Ix  i  0  ;  the  first  n  elements  of  the  slack -vector  are  then  exactly 
the  elements  of  x  . 

In  Theorems  1  and  2  we  will  derive  the  Kuhn -Tucker  conditions. 
Consider: 

(1.4)  Q(x,  u)  =  a‘x  -  -^x'Bx  -  u'[C*x  +  Iw  -  d]  , 

where  the  m  -vector  u  will  be  called  the  Lagrangean  vector.  Then: 

Theorem  1 :  A  sufficient  condition  for  x  to  solve  the  problem 
(1.1)  -  (1.2)  is: 

i)  C‘x  ^d  [or,  equivalently,  C'x  +  Iw  =d;  w  i  0]  , 

ii)  ^1  ♦  =  0  [  1.  e.  a  -  Bx*  -  Cu*  =  0 1 

'  dx  x=y  ••  ^ 

iii)  u*  i  0 

iv)  u  'w  =  0  . 

Proof :  Suppose  x  is  a  vector  such  that  C'x  £  d  (or 
C'x+Iw=d,  w20).  Then: 
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(1.5)  Q*(x*)  -Q*(x)  =a'(x*-x)  -  ix**Bx*  +  Y  x'Bx 


*  —  1_*  —  *  «4c4c_ 

=  a'(x  -  x)  +  — (x-x  )'B(x-x  )-x  'Bx  +  x  'Bx 


=  (a'  -  x*'B)(x*  -  X  )  +  -^  (x-  X*)  'B(  x-  x* ) 
=  u^'C'Cx*  -  X  )  +  (  X  -  X*)  'B(  x-  X*) 


=  u’*''(d  -  C'x)  +  Y  (  X  -  X*)  'B(  X  -  X*) 

=  u*'  w  +  (  X  -  x*)B{  X-  X*)  >  0  , 

♦  — 

since  u  ^  0 ,  w  i  0 ,  and  B  is  positive  definite.  The  equality  only  holds 

,  —  ♦ 
if  X  =  X  . 

4< 

Theorem  2;  A  necessary  condition  for  x  to  solve  (1.11  -  (1.2) 
is 

i)  C'x  ^  d  [or,  equivalently,  C'x  +Iw  =d,  w  >0] 

ii)  a-Bx*-Cu*=o] 

iii)  u*  ^  0 

iv)  u  '  w  =  0  . 
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Proof  ;  i)  is  obvious,  ii)  is  the  well-known  Lagrangean 
condition  for  maximizing  under  constraints.  As  for  iii)  ,  consider 


(1.6) 


which  implies  that,  if  u  has  a  negative  element,  a  decrease  in  its 
associated  source  d  (or,  equivalently,  an  increase  in  the  associated 
slack  w)  will  increase  the  value  of  Q  =  Q  .  There  is  nothing  in  the 
nature  of  the  relevant  inequality  of  (1.2)  preventing  this  decrease. 
Hence,  no  element  of  u  can  be  negative^  .  To  prove  iv) ,  suppose 
u^>0.  Then,  vide  (1.6),  an  increase  in  d^,  which  is  feasible 
as  long  as  Increases  Q.  Hence,  if  Uj^>0,  Wj^>0, 

there  is  no  maximum.  Conversely,  suppose  w^  is  positive.  Then, 
if  Uj^  >  0 ,  since 


(1.7) 


a  decrease  in  w^^  Increases  Q .  Hence,  no  maximum. 

The  two  theorems  lead  to  the  following  approach.  From 


(1.8) 


a  -  Bx  -  Cu  =  0 


^  Actually,  this  argument  glosses  over  a  subtle  point.  For  it  requires  the 
vector  u  to  be  defined!  Using  (1.10)  below  we  see  that 

u  =  (C  b'^C  )"^d  -  C*  B‘^a)  , 
w  w  '  w  * 

where  C  is  the  submatrix  of  C  consisting  of  all  rows  with  slack  zero, 
w 

If  the  rows  of  C'  are  dependent,  u  cannot  be  uniquely  solved.  In  some 
w 

cases,  this  can  indeed  lead  to  difficulties. 
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and 

(1.9  C'x  +  Iw  =  d 

we  derive 

(1.10)  -C‘B"^Cu  +  Iw  =  d  -  C'B“^a 

by  solving  (1.8)  for  x  and  substituting  the  result  in  (1.9) .  The  problem 
is  to  find  a  non-negative  solution  (u  ,  w  )  to  this  system  of  m  equations 
in  2m  unknowns  such  that  u**w*  =  0 .  As  initial  solution  we  can  clearly 
take^  w  =  d  -  C*B"^a.  If  w  >  0  ,  then  we  have  the  solution.  If  not,  we 
replace  a  negative  w^  by  u^  (which  u^  will  then  be  positive,  see  below). 
Thus,  we  invariably  fulfill  the  requirement  u*w  =  0  . 

If,  at  any  stage,  a  w^  is  negative,  this  implies  that  the  i^^  constraint 
is  violated.  In  the  next  step,  if  we  impose  ^  0 ,  we  maximize  a*x  -  ~  x*Bx 
subject  to  the  same  constraints  as  before  plus  the  i^^  constraint.  This  vlll 
decrease  the  value  of  Q  . 

If,  however,  at  any  stage  u^  is  negative  (hence  w^  =  0) ,  then  the 

i^^  constrained  is  imposed  (cjx  =  d^) .  In  the  next  step,  putting  =  0  , 

we  maximize  a'x  -  y  x'Bx  subject  to  the  same  constraints  as  before,  minus 
th 

the  i  constraint.  This  will  increase  the  value  of  Q  . 

^  If  there  are  n  non-negativity  conditions  the  first  n  elements  of  w  are  the 
values  of  x .  These  values  are 

X  =  0  -  (-I)*B“^a  =  B"^a  , 

the  vector  of  the  unconstrained  maximum. 
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Variables  imposed  to  be  zero  are  called  non-basic  variables.  The 
other  variables  are  called  basic  variables.  In  the  initial  solution  the 
(i  =1,  ... ,  m)  are  non-basic,  the  (i  =  1>  ••• ,  m)  are  basic. 

Throughout  all  stages  of  the  solution  process,  if  u^  is  basic,  then 
is  non-basic  and  vice  versa. 

Theorem  3  ;  Consider  a  solution  (u,  w)  to  (2,10)  satisfying 
u*  w  =  0 .  Then,  if  u^  =  0 ,  <  0  a  switch  making  u^  basic  and 

non-basic  will  lead  to  u^  >  0 .  Conversely,  if  u^  <  0 ,  w^  =  0 ,  a 
switch  making  u^  non-basic  will  lead  to  w^  >  0 . 

Proof  ;  If  u^  <  0 ,  w^  =  0  the  switch  will  increase  the  value  of 
Q,  because  we  are  maximizing  subject  to  one  constraint  less.  Since 

(l.U) 

increasing  Q  implies  decreasing  d^,  or  increasing  w^  (from  0  to  some 
positive  value)  .  Again,  if  w^  <  0 ,  u^  &  0 ,  a  switch  will  decrease  the  value 
of  Q  ,  because  we  are  maximizing  subject  to  one  constraint  more.  Now 


(1.12) 


An  increase  in  w^  (from  a  negative  value  to  0)  decreases  Q,  hence 
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increases  the  value  of  (from  0  to  some  positive  value)  ^  . 

The  question  which  or  Uj^,  If  negative,  to  replace  Is  of 
progmatic  interest.  The  most  natural  approach  appears  to  be  to  replace 
the  most  negative  value.  An  alternative  procedure  is  to  consider  all 
quotients  (all  w^|wj<0)  and  (all 

switch  the  variables  with  the  largest  ratio.  Neither  of  these  procedures 
is  foolproof  against  cycling,  but  for  practical  pu  rposes  this  is  of  no 
consequence.  For  theoretical  purposes,  switching  procedures  can  be 
(and  have  been)  constructed  which  exclude  cycling  altogether,  cf.  [1]  . 
An  Example 

2  12 

Maximize  3Xj  +  4x^  -  3Xj  -  4x^X2  -  I'J^  ^2 

Subject  to  -Xj  <  0 

-X2<0 
^1  +  2X2  <  4 


^  A  little  algebra  shows  that  specifically  if  u.  <  0 ,  w  =  0 ,  then  the  switch 

1  ^  ^  th 

leads  to  a  value  of  w^  equal  to  "  ~  >  where  p  is  the  ( h,  h)  diagonal 

element  of  the  inverse  of  the  positive  definite  matrix  C  B~^C  if  c,  is  the 
th  w  w  i 

h^  row  of  C'  ,  cf.  footnote  1.  Conversely,  if  w  <  0,  u,  =  0  the  value 

^  ^  th  * 

of  u^  after  the  switch  is  -v  w^  ,  where  0-  is  the  (k,  k)  diagonal  element 

of  the  inverse  of  C  B”^C  ,  if  the  newly  included  c,  is  the  k^^  row  of  C  . 

w  w  i  w 
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Hence, 


Hence,  consider: 


Basic 

Value 

“2 

“3 

^2 

^^3 

• 

-4 

2 

■4 

1 

0 

0 

• 

6 

2 

-3 

4 

0 

1 

0 

^3 

-4 

-4 

4 

-4 

0 

0 

1 
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Switch  Wj  and 


Basic 

Value 

^^2 

^^2 

^^3 

1 

11 

5 

u. 

4 

1 

0 

0 

1 

“  2 

4 

4 

w^ 

0 

0 

0 

1 

1_ 

2 

2 

2 

1 

5 

3 

u 

-1 

0 

1 

0 

3 

'  2 

4 

~  4 

Switch 

u^  and  w^  ; 

^  1 

4 

5 

2 

u. 

2- 

1 

0 

0 

1 

3 

"  3 

3 

■  3 

,  1 

1 

2 

4 

w. 

1- 

0 

1 

0 

2 

3 

■  3 

3 

3 

,  1 

2 

4 

5 

w. 

1- 

0 

0 

1 

3 

3 

3 

"  3 

"  3 

This  is  the  solution.  Knowing  the  vector  w  we  can  easily  solve 
C'x  -I-  w  =  d;  in  fact,  when  there  are  non -negativity  conditions  the  first  n 
elements  of  w  coincide  with  those  of  x ;  hence  x  =  0 ,  x  =  1  •  We 

would  have  gotten  the  final  tableau  directly  by  switching  w^  and  Ui  ^ 

Wi  w^  J 

the  first  tableau  ( —  >  —  )  .  We  also  have  u,  =  2—  ,  u_  =  0,  u*  =  0  . 

Vlj  ^3  1  V  M  3 

These  values  of  the  Lagrangeans  are  an  Indication  of  the  value  of 

the  sources,  by  virtue  of  (1.  6) .  Moreover,  they  are  the  solution  to  the 
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so-called  dual  problem.  Formally,  if  the  primal  problem  is  given  by 
(1.1)  -(1.2),  and  has  a  solution  x  ,  say,  then  the  m -vector  u 
solves  the  problem 

(1.13)  Minimize  d*u  +  Bx* 
subject  to 

(1.14)  u>0 

and 

(1.15)  Cu  +  Bx"^  =  a  . 


Moreover: 

4<14>  e  eiee 

(1.16)  a'x  -  yx  'Bx  =d'u  +  jx  'Bx  . 


Relations  (1.14)  -(1.16)  can  be  verified  numerically. 
The  example  is  illustrated  in  Figure  1 . 


Figur*  1 


2.  linear  Programming 


The  easy  procedure  outlined  above  breaks  down  when  B  is  semi- 
definite.  For  then  C'B~^C  and  C'B~^a  cannot  be  determined.  Consider 
first  the  most  extreme  case,  where  the  B  -matrix  is  the  null-matrix,  and 
hence  we  have  a  linear  programming  problem.  We  can  then  apply  the 
procedure  above  by  introducing  a  large  value  X.  and  considering  the 
problem: 

(2.1)  Maximize  X(a*x)  -  '|'X'Ix 
subject  to 

(2.2)  C'x<d  , 

which  is  clearly  equivalent  to  the  linear  programming  problem  of  maximizing 
a'x  subject  to  C'x  <  d  . 

As  an  illustration  consider  the  problem: 

Maximize  L(x)  =  3Xj  +  ux^  [or  f(Xx)  =  X (3Xj  +  4X2) --jx'Ix] 

-Xj  s  0 


Subject  to 
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We  obtain,  since  B  =  =  I  : 


which  leads  to  the  following  tableaux: 

I 


Basic 

Value 

“l 

“2 

^^3 

"4 

“5 

^1 

^"2 

^3 

^4 

^5 

^1 

0  + 

3\ 

-1 

0 

1 

2 

1 

1 

0 

0 

0 

0 

^2 

0  + 

4X 

0 

-1 

-  2 

-3 

4 

0 

1 

0 

0 

0 

^3 

1  + 

5X 

1 

-2 

-5 

-8 

7 

0 

0 

1 

0 

0 

^4 

2  + 

6X 

2 

-3 

-8 

-13 

10 

0 

0 

0 

1 

0 

3  - 

19X 

1 

4 

7 

10 

-17 

0 

0 

0 

0 

1 

II  (Switch  w  and  u. ;  divide  all  entries  by  17) 
D  5 
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The  solution  therefore  is  :  >  ''^2  “  *2  “  ’  ^3  "  IT  ’ 

67 

=  Wg  =  0 ,  <»  =  YY  •  The  solution  is  illustrated  in  Figure  2.  The  first 
tableau  gives  as  solution  ( 3X ,  4X ) ,  i.  e.  a  point  on  the  gradient  line  to 


3x^  +  4X2  ,  the  linear  part  of  the  objective  function.  Since  this  point  violates 
the  5^^  constraint,  we  next  maximize  3Xx^ -I- 4XX2  -  subject  to 

x^  4  4X2  =  3 .  Because  maximizing  3Xx^  4  4Xx2  -  is  the  same  as 


minimizing 


we  actually  minimize  the 


distance  between  ( 3X,  4X )  and  the  line  x^  4  4X2  =  3  ;  i.  e.  we  find  the 


projection  of  ( 3X,  4X )  on  x^  4  4X2  =  3 .  In  the  next  tableau  we  find  the 
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solution.  Either  criterion  tells  us  to  switch  u .  with  w .  . 

4  4 

It  is  of  some  interest  to  notice  that  it  will  take  at  least  n  steps  to 

get  to  the  solution  point,  since  at  least  n  elements  of  w  will  be  zero  in 

the  solution  -  barring  infinite  maxima.  It  may  also  be  observed  that  the 

solution  of  the  dual  is  given  by  the  coefficients  of  X  in  the  final  tableau, 

8  17 

1. e.  Uj  =  0,  u^  =  0,  u^  =  0,  =  YY  »  “5  “  YY  •  follows,  because 

d^(Xx)  /dx  =  Xu. 
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3.  Singular  Quadratic  Programming 

We  are  now  in  a  position  to  consider  the  more  general  case  of 
quadratic  programming  with  a  singular  quadratic  part.  Quite  generally, 
if  B  is  a  positive  semi-definite  matrix  of  order  n  X  n  and  rank  r  there  exists 
an  nXr  matrix  T'  such  that  T'T=B.  Introducing  the  transformation 

=  T  X 

[where  is  a  r -vector,  y^  a  (n-r) -vector  and  E  a  matrix  which 

has  as  row  the  (r  +  k)  n-dimensional  unit  vector]  we  can  therefore 
write,  provided  we  take  care  that  the  first  r  columns  of  T  are  independent; 

(3.2)  a*x  -  -^x'Bx  =  a‘T*”V  -  * 

♦-1  1 

Maximizing  a’T  V  ~  is  equivalent  to  maximizing 

provided  \  is  very  large. 


(3.1) 


y  = 


^1 

= 

Tx 

^2. 

= 

Ex 
»  _ 

As  an  example  consider  the  problem 
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•  ■ 

•  • 

Maximize  Sx^  +  4X2  +  j  (x^  X2  x^) 

1  2  -1 

^1 

2  4-2 

^2 

-1  -2  1 

Subject  to  -x^  ^  0 

-X2  =  0 

-Xj  ^  0 

Xj  +  2X2  +  X3  ^  4 


Using 


We  reformulate  this  problem  as  follows. 

Maximize  x( 3yj  -  Zy^  +  4yj  -  iy^)  -  iy^  -  iy^ 

Subject  to  -Yj  +  Zy^  -  S  0 

-y2  ^0 

-YjSO 

Yj  +  Zy^  S  4 
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With 


" 

— 

• 

m 

•  « 

« 

r  1 

-1 

2 

-1 

B  = 

\ 

0 

0 

d  = 

0 

a  = 

3X  1 

1 

0 

-1 

0 

0 

1 

0 

0 

-2X 

0 

0 

-1 

0 

0 

1 

0 

4X 

- 

- 

_  1 

0 

2 

4 

m  m 

we  obtain 


c»b'^c  = 

5X  +  1 

o 

-i 

■ 

2X  -I-  1 

* 

d-C‘B  a  = 

3  +  8X 

X 

£ 

X 

2 

-1 

0 

0 

-2X 

-1 

0 

-1 

2 

4X 

2X-I-1 

X 

0 

2 

4X  -1- 1 

X 

1 

00 

1 

1 

Thus,  we  get  the  following  tableaux: 

I 


Basis 

Value 

^^2 

% 

W 

2 

^3 

^4 

^1 

5X  +  1 
X 

2 

-1 

2X  4  1 

X 

1 

0 

0 

0 

-  2X 

2 

-1 

0 

0 

0 

0 

0 

0 

^3 

4X 

-1 

0 

-1 

2 

0 

0 

0 

1 

^4 

1  -8X 

2X4  1 

X 

0 

2 

4X  4  1 

X 

0 

0 

0 

1 
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II  (With,  for  reasons  of  simplicity,  replacing  w^) 


Basis 

Value 

“l 

“2 

“3 

^1 

^2 

^3 

^^4 

^1 

3  +  4X 

X  +  1 

X 

0 

-1 

2X  4  1 

X 

1 

2 

0 

0 

^2 

2X. 

1 

-2 

1 

0 

0 

0 

-1 

0 

0 

^3 

4X 

-1 

0 

-1 

2 

0 

0 

1 

0 

1 

00 

2X  +l 

X 

0 

2 

4X  4  1 

X 

0 

0 

0 

1 

1 

III  ( Switching  u^ 

and  w.) 
4 

lOX  +4 

1 

A 

1 

A 

1 

A 

2X  4  1 

4X  +  1 

"4X  +1 

U 

4X  4  1 

U 

L 

U 

4X  4  1 

2X 

-2 

1 

0 

0 

0 

-1 

1 

0 

^3 

6\ 

1 

n 

1 

A 

A 

A 

A 

2X 

4X  +  1 

4X  +  1 

4X  4  1 

U 

U 

V 

U 

4X  4  1 

“4 

X(8X  -  1) 

2X  4  1 

o 

2X 

1 

A 

A 

A 

X 

4X  +  1 

4X  41 

V 

4X  41 

V 

V 

U 

4X  + 1 

In  the  first  tableau  we  have 

^2 

=  w2  =  -2X;  yj  = 

W3  = 

4X  ;  yj 

=  272- 

yj  +  wj  = 

1  2 

This  is  clearly  the  solution  maximizing  3y^  -  -i-  4y^  ~  7^1  *  However,  we 

violate  the  2^^  and  4^^  constraint  (as  indicated  by  the  negative  values  for 
w^  and  w^ ).  First  imposing  =  0  and  next  w^  =  0  leads  to  the  final 
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tableau.  We  have  =  0 ,  and  =  0  . 

Hence  72  =  0>  iTTT  ~  ^1  ""  4X  1 1  ^  ®  check, 

y^  +  2y^  =  4.  Transforming  back  to  the  original  variables  we  find  =  2-|  , 

X-  =  0,  X,  =  l4  )  and  the  value  of  the  objective  function  equals  8-^  . 

2  3  2  6 

Again,  the  coefficients  of  X  give  the  dual  values;  hence  u^  =  0  ,  =  2, 

U3  =  0 ,  u^  =  ~  I  =  2  .  Checking  ( 1. 15)  we  find : 


•  m 

•  « 

■k  * 

•  « 

•  • 

-10  0  1 

0 

1  0  0 

1 

3 

2-100 

2 

+ 

0  0  0 

0 

= 

-2 

1 

0 

1 

IV 

0 

0  0  0 

0 

4 

L  J 

-  . 

.  . 

2 

»  1 

Writing  i  ( X )  for  linear  functions  in  X  ,  and  q(  X )  for  quadratic  functions 
of  X  ,  any  problem  with  a  finite  solution  will  have  forms  of  the  structure 


ifCX) 

for  all  basic  w  ,  and  "  |  for  basic  u  .  The  expressions  in 

1  X\K  )  1 


the  tableaux  themselves  will  invariably  be  of  the  form 


ijiX) 

T^)  • 


Again,  we  can  make  the  observation  that,  if  there  is  a  finite  solution, 
at  least  n  -  r  steps  will  be  required.  For; 


Theorem  4 ;  If  the  maximum  of  a'x  -  ’^x'Bx  ,  (where  B  is  of  order 

nXn  and  rank  r  ),  subject  to  C'x  <  d  is  finite,  then  at  least  n-r  constraints 


T 
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will  be  exactly  satisfied. 

Proof;  Rewrite  so  as  to  get 

4<-l  1 

Maximize  a*T  y- 

Subject  to  C’T*  V  =  d  • 

The  r  variables  y^  take  finite  values.  For  any  set  of  values,  say 

y^^ ,  there  remains  a  linear  programming  problem  in  n  -  r  variables.  Hence 

at  least  n  -  r  constraints  will  be  binding. 

If  we  can  find  n  -  r  constraints  binding  in  the  solution  point  by 

inspection ,  the  problem  can  immediately  be  transformed  to  a  non- singular 

quadratic  programming  problem.  For  maximizing  a'x  -  “^x'Bx  (B  is  of 

order  nXn,  rank  r)  subject  to  C'x  =  d  (where  C*  has  at  least  n-r 

rows  c*  )  is  equivalent  to  maximizing 
n 

a'x  -  4  x*[  B  +  c  ,  c  *  +  ...  c  c '  lx 
2  *■  11  n-r  n-r-* 

subject  to  C'x  =  d.  The  matrix  B  c  ,c -i-  c  c'  will  be  of  full 

11  n-r  n-r 

is  of  full  rank. 


rank  iff  the  matrix 
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